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SUMMARY 


Introduction.  This  report  describes  the  development, 
validation,  and  utilization  of  a multi-attribute  utility 
model  and  a probabilistic  cost  model  for  use  by  the  SINCGARS 
Special  Task  Force  (STF)  in  evaluating  the  military  utility 
and  cost  of  a series  of  alternative  radio  systems.  These 
models  were  programmed  to  be  interactive  and  were  made 
available  to  the  SINCGARS  STF  in  order  to  permit  additional 
analyses.  A separate  report  w an  submitted  to  the  SINCGARS 
STF  on  the.  development  of  these  models.  The  major  difference 
between  that  report  and  this  report  was  the  inclusion  of 
appendices  in  the  SINCGARS  STF  report  providing  more  detailed 
information  with  respect  to  such  factors  as  utility  functions 
importance  weights,  values  of  system  parameters,  and  techni- 
cal performance  characteristics.  •/ 

Background  and  Approach.  The  SINCGARS  Special  Task 
Force  was  established  by  the  Army  Chief  of  Staff  for  the 
purpose  of  defining  and  recommending  an  optimum  solution  for 
meeting  the  Army's  requirement  to  ensure  that  a prescribed 
level  of  suitable  Very  High  Frequency  (VHF) -Frequency 
Modulated  (FM)  Combat  Net  Radios  (CNRs)  will  be  available  in 
the  Army  inventory  in  the  post- 1980  time  frame.  To  this 
end,  the  SINCGARS  STF  defined  the  following  four  alternative 
courses  of  action  for  comparison/evaluation  with  respect  to 
satisfying  the  Army's  requirements  for  CNRs: 

• Alternative  1 . Retain  the  current  VHF-FM  CNRs 
(which  consist  of  three  families  of  radios)  with 
minor  product  improvements. 

• Alternative  2.  Develop  and  ultimately  procure  a 
product-improved  version  of  the  current  VHF-FM 
combat  net  radios  that  will  operate  with  a 25  KHz 
channel  spacing. 

• Alternative  3 . Continue  development  and  ultimately 
procure  Radio  Set,  AN/URC-78. 

• Alternative  4.  Develop  and  ultimately  procure  a 
new  family  of  VHF-FM  combat  net  radios. 

The  objectives  of  the  research  described  in  this  report 
were  to: 

• Develop  and  test  a multi-attribute  utility  model 
to  be  used  to  assist  the  SINCGARS  STF  of  the  U.S. 
Army  in  evaluating  the  military  utility  of  alter- 
native Combat  Net  Radio  configurations. 


• Evaluate  the  hardware  cost  estimates  of  Alterna- 
tives 2,  3,  and  4 utilizing  an  interactive  Monte 
Carlo  program. 

Of  particular  importance  during  the  development  of  the 
multi-attribute  utility  model  was  the  need  to  provide  an 
analytic  tool  which  would  accept  "last  minute"  changes  in 
data  and  would  also  enable  a user  to  conduct  "what  if" 
analyses  on  an  interactive  basis.  A multi-attribute  utility 
model  was  developed  which  would  systematically  quantify  the 
utility  of  alternative  radio  systems.  The  model  utilizes 
as  inputs  the  technical  performance  characteristics  of  a 
radio  system.  A hierarchical  structure  consisting  of  several 
levels  was  developed,  starting  with  military  utility  and 
partitioning  this  into  major  dimensions  of  utility,  e.g., 
technical  system  utility  and  operational  acceptability. 

These  dimensions  were  further  sub-divided  into  sub-dimen- 
sions which,  in  turn,  were  further  partitioned--each  parti- 
tion becoming  more  specific  until  a level  was  reached  at 
which  one  or  more  technical  performance  characteristics 
served  to  describe  each  of  the  sub-dimensions.  The  military 
utility  for  different  levels  of  each  of  the  performance 
characteristics  was  established  by  assessing  a utility 
function  over  the  relevant  range  of  that  characteristic. 

The  relative  importances  of  the  different  performance  charac- 
teristics were  assessed  by  assigning  relative  importance 
weights.  This  assignment  of  weights  was  accomplished  for 
the  components  of  each  level  of  the  hierarchy.  The  proce- 
dures for  assignment  depend  upon  how  the  components  combine 
to  determine  system  utility  at  the  level  in  question. 
Combination  rules  are  additive  for  components  which  inde- 
pendently contribute  to  the  utility  at  a given  level  and  are 
multiplicative,  if  the  contribution  of  one  or  more  com- 
ponents at  a given  level  depends  upon  the  value  of  one  or 
more  other  components  at  that  level. 

The  resultant  structure  thus  systematically  combined 
expert  judgment  and  technical  performance  characteristics  to 
provide  a model  which  accurately  aggregated  the  actual 
measures  of  the  performance  characteristics  of  a particular 
system  so  as  to  yield  a measure  of  the  military  utility  of 
the  system. 

The  resultant  model  was  subjected  to  both  internal  and 
external  sensitivity  analyses  to  ensure  insensitivity  to 
minor  variations  in  weights  and  sensitivity  to  different 
alternative  systems.  The  model  was  then  employed  to  evaluate 
the  four  alternative  radio  systems.  Also  evaluated  was  the 
Korean  War  System  (KWS) , which  served  as  a point  of  reference 
for  comparison  of  utilities. 

In  the  evaluation  of  the  15-year  life  cycle  cost  esti- 
mates of  Alternatives  2,  3,  and  4,  the  purpose  of  the  Monte 


Carlo  program  which  was  developed  and  used  was  to  combine 
the  probability  distributions  for  the  dollars  of  average  unit 
of  cost  ($AUC)  of  each  radio  and  its  ancillary  component 
for  each  platform  (manpeick,  vehicular,  and  airborne)  in 
proportion  to  the  Authorised  Acquisition  Objectives  (A AOs) 
of  those  radios  and  components.  The  outouts  of  this  Monte 
Carlo  program  were: 


• A probability  distribution  for  the  $AUC  of  each 
alternative  radio  system,  by  platform. 

• A probability  distribution  for  the  $AUC  of  each  of 
the  radios  and  their  ancillary  components  by 
platform. 

Technical  Implications.  Having  structured  and  tested 
the  SINCGARS  Evaluation  Model  as  previously  described,  the 
four  alternative  systems  and  the  Korean  War  System  were 
evaluated.  The  results  are  summarized  in  the  table  below. 


A SUMMARY  OF  THE  MILITARY  VALUES  OF 
OF  ALTERNATIVE  RADIO  SYSTEM  CONFIGURATIONS 


Alternative  Radio  system.  Configurations 


Major  Dimensions  of  Value 


Operational  Acceptability 


Technical  System  Utility 


Overall  Military  Value 


Korean 

War 

System 
( KWS ) 


Current 

Product  - 

System. 

Improved 

(Alt . 1) 

System 
(Aft.  21 

1 

. 69  1 

mm 

. 34 

. 39 

. 51 

.56 

mental 
System 
OUt.  31 


System 
(Alt.  41 


*Assuming  that  operational  acceptability  and  technical  system 
utility  are  weighted  equally  in  determining  overall  military 
value . 


The  results  indicate  that  Alternatives  3 and  4 are 
superior  to  Alternatives  1 and  2 in  terms  of  military  value 
or  benefit,  mainly  due  to  differences  in  technical  system 
utility.  Further  examination  of  the  results  indicated  that 
this  superiority  with  respect  tc  technical  system  utility  is 
due  to  a significant  improvement  with  respect  to  both  depen- 
dability and  technical  performance.  Technical  system  utility 
is  a multiplicative  combination  of  these  two  factors. 
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Alternatives  1 and  2 are  both  far  superior  to  the 
Korean  War  System  (KWS)  ; however,  Alt  ex-native  2 is  only 
slightly  superior  to  Alternative  1 with  respect  to  technical 
system  utility  and  operational  acceptability. 

With  respect  to  the  Monte  Carlo  evaluation  of  hardware 
cost  estimates,  a summary  of  the  most  likely  $AUC  for  the 
three  alternative  systems  by  platform  is  provided  in  the 
table  which  follows: 


A SUMMARY  OF  THE  MOST  LIKELY 
SAUC  BY  ALTERNATIVE  AND  PLATFORM 

Most  Likely 

$AUC  of 

Alternative  Systems  j 

Platform 

2 

3 

4 

Manpack 

$1090 

$1590 

$1550 

Vehicular 

3190 

2980 

2570 

Airborne 

3980 

1990 

1970 

Combining  the  foregoing  results  of  the  evaluation  of 
the  military  utility  or  benefit  of  the  alternative  radio 
systems  with  the  results  of  the  evaluation  of  hardware  cost 
estimates,  the  following  conclusions  were  reached  with 
regard  to  the  relative  cost/benefit  merits  of  the  alternative 
systems : 

• Inasmuch  as  Alternative  2 was  far  more  expensive 
than  either  Alternative  3 or  Alternative  4,  the 
latter  two  alternatives  a1- a better  options  from  a 
combined  cost/benefit  point  of  view. 

• Because  Alternative  4 has  a lower  hardware  cost 
than  Alternative  3,  and  the  military  values  of 
the  utilities  of  Alternative  3 and  4 are  approx- 
imately equal,  Alternative  4 is  considered  to  be 
>-na  most  cost/beneficial  option.  However,  this 
conclusion  must  be  qualified  somewhat  by  the  fact 
that  the  employment  of  two  receiver/transmitters 
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with  Alternative  3,  rather  than  an  auxiliary 
receiver,  increases  its  average  unit  cost  ($AUC) 
by  about  $500.00.  If  this  situation  were  rec- 
tified, Alternative  3 would  compare  quite  favorably 
with  Alternative  4 from  a cost/benefit  standpoint. 

The  foregoing  results  were  based  upon  the  information 
which  was  available  with  regard  to  the  various  alternative 
radio  systems.  The  flexible  structure  of  the  SINCGARS 
Evaluation  Model,  however,  he  s enabled  the  SINCGARS  STF  to 
continue  to  use  the  model: 

• With  new  or  modified  input  data. 

• To  conduct  "what  if"  analyses  on  an  iterative, 
interactive  basis  in  response  to  queries  that  have 
been  received. 

In  so  doing,  the  SINCGARS  STf  has  attained  an  increasingly 
better  understanding  of  all  of  the  factors  involved  in  the 
evaluation  of  the  alternative  systems  and  a greater  degree 
of  confidence  in  the  recommendations  which  they  have  made 
based  upon  their  use  of  the  model. 

Methodological  Implications.  The  following  two  addi- 
tional  observations  are  also  important  from  the  standpoint 
of  decision  analytic  theory: 

• The  evaluation  of  alternative  systems  is  very 
closely  related  to  system  design,  so  the  evalua- 
tion model  must  be  very  flexibly  structured  with 
respect  to  possible  changes  in  inputs  in  order  to 
permit  meaningful  "what  if"  analyses.  For  exam- 
ple, the  range  of  a technical  performance  charac- 
teristic over  which  a utility  function  is  assessed 
should  not  be  the  entire  physical  performance 
range  of  a general  system,  for  this  will  cause  the 
model  to  be  quite  insensitive.  If  specific  alter- 
natives which  will  not  change  are  to  be  evaluated 
for  purpose  of  choosing  among  them,  the  range  of 

a performance  characteristic  should  be  the  ran^e 
for  the  alternatives  under  consideration.  However, 
when  the  designs  of  developmental  or  conceptual 
systems  are  being  evaluated,  then  the  range  of  a 
technical  performance  characteristic  over  which  a 
utility  function  is  assessed  should  reflect  the 
minimum  feasible  to  the  maximum  feasible  performance 
range  of  those  systems. 

• Close  attention  must  also  be  given  to  the  defini- 
tion of  scenarios  as  conditioning  variables  in  the 
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evaluation  of  alternative  systems.  The  scenarios 
should  be  so  defined  that  not  only  will  they  be 
representative  of  the  situations  in  which  the 
alternatives  will  operate  and  thus  permit  the 
assessment  of  the  expected  utilities  of  all  of  the 
alternative  systems,  but  also  they  must  serve  to 
discriminate  clearly  among  the  alternative  systems. 
These  goals  are  not  usually  compatible,  and  an  in- 
depth  analysis  of  which  goal  to  pursue  must  be 
accomplished  in  order  to  decide  upon  the  defini- 
tion of  scenarios. 
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THT  DEVELOPMENT  OF  A SINGLE  CHANNEL  GROUND  AND 
A I RT;  NE  RADIO  SYSTEM  (SINCGARS)  EVALUATION  MODEL 


1.0  INTRODUCTION 

The  Single  Channel  Ground  and  Airborne  Radio  System 
(SINCGARS)  Special  Task  Force  (STF)  was  established  on 
December  2,  1974  in  accordance  with  the  Army  Chief  of  Staff's 
Memorandum  (CSM)  74-15-101,  dated  November  21,  1974.  The 
charter  for  the  SINCGARS  STF: 

• Defines  SINCGARS  "as  a radio  communication  system 
designed  primarily  to  be  organic  to  and  operated 
by  the  combat  arms.  The  system  will  be  used  by 
the  Army  in  the  field  where  i£s  employment  is 
required  and  cost  effective." 

• Describes  the  mission  of  the  SINCGARS  STF  in  terms 
of  establishing  "the  management  and  technical 
base,  under  the  provisions  of  Army  Regulation 
1000-1,  for  a cohesive,  integrated,  secure  com- 
munications system  as  specified  in  the  approved 
Required  Operational  Capability  (ROC)  for  the 
VHF-FM  portion  of  the  SINCGARS.  This  effort  will 
consider  the  impact  or  relationship  to  all  single 
channel  communications  systems  designated  to  be 
organic  to  and  operated  by  the  combat  arms  in  the 
post-1980  time  frame." 

Within  the  context  of  this  mission  statement,  the 
objective  of  the  SINCGARS  STF  may  be  further  described  in 
terms  of  defining  and  recommending  an  optimum  solution  for 
meeting  the  Army's  requirement  to  ensure  that  a prescribed 
level  of  suitable  combat  net  radios  (CNRs)  will  be  available 
in  the  Army  inventory  in  the  post-1980  time  frame.  To  this 
end,  the  SINCGARS  STF  identified  the  following  four  alter- 
native courses  of  action  with  respect  to  satisfying  the 
Army's  requirements  for  CNRs: 

• Alternative  1 . Retain  the  current  VHF-FM  CNRs 
(which  consist  of  three  families  of  radios)  with 
minor  product  improvements. 

• Alternative  2.  Develop  and  ultimately  procure  a 
product- improved  version  of  the  current  VHF-FM 
combat  net  radios  that  will  operate  with  a 25  KHz 
channel  spacing. 

• Alternative  3 . Continue  to  develop  and  ultimately 
procure  Radio  Set,  AN/URC-78. 

^Charter  for  the  Department  of  the  Army  Single  Channel  Ground 
and  Airborne  Radio  Subsystem  (SINCGARS)  Special  Task  Force 
(STF) , Office  of  the  Chief  of  Staff,  Department  of  the  Army, 
February  2n,  1975. 
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Alternative  4.  Develop  and  ultimately  procure  a 
new  family  of  VHF-FM  combat  net  radios. 


In  order  to  obtain  a better  understanding  of  the  nature 
of  the  foregoing  alternatives,  it  is  desirable  to  know  more 
about  the  types  of  radios  involved  in  each  alternative. 

Based  upon  information  provided  by  the  SINCGARS  STF,  the 
types  and  general  quantities  of  radios  may  be  summarized  as 
follows : 

• Alternative  1:  Current  Series  of  CNRs.  As  pre- 

viously indicated^  the  current  series  of  CNRs 
actually  consists  of  three  more  or  less  inde- 
pendent families  of  radios,  which  are  the  AN/PRC- 
77  (a  manpack  radio) , the  AN/VRC-12  (a  vehicular 
radio) , and  the  AN/ARC-114  (an  airborne  radio) . 
Although  these  families  of  radios  all  operate  in 
the  30-76  MHz  frequency  band,  they  have  been 
developed  independently  to  satisfy  specific  user 
needs  for  manpack,  vehicular  and  airborne  com- 
munications. The  radios  have  been  modularly 
constructed  to  allow  for  improved  maintenance  and 
growth  capabilities,  but  no  commonality  of  modules 
exists  among  these  radio  families.  The  three 
families  of  radios  may  be  further  described  as 
follows : 

AN/PRC-77 . This  manpack  radio  provides  a 
short-range,  frequency  modulated  (FM)  voice 
communications  capability  for  the  Army  in  the 
field.  The  AN/PRC-77  became  operational  in 
1968  as  a result  of  a product  improvement 
program  of  the  AN/PRC-25.  Three  omni-direc- 
tional antennas  can  be  used  with  the  AN/PRC- 
77;  namely,  a 3-foot  semi-rigid  flexible 
ste.  ' tape  antenna  (AT-892/PRC) , a 10-foot 
foldable  antenna  (AT-271A/PRC) , and  a tele- 
scoping 3 to  10-foot  antenna  (AS-2109) . With 
the  addition  of  an  Amplifier  Power  Supply 
Group  (OA-3633) , Vehicular  Mounting  (MT- 
1029),  and  Vehicular  Antenna  (AS-1729) , the 
AN/PRC-77  can  be  configured  for  vehicular 
use.  As  such,  it  is  redesignated  the  AN/VRC- 
64.  A configuration  which  allows  for  both 
manpack  and  vehicular  operation  is  the 
AN/GRC-160.  A more  detailed  description  of 
uhe  technical  characteristics  of  this  manpack 
radio  is  provided  in  Appendix  E. 

AN/VRC-12 . This  family  of  vehicular  radios 
became  operational  in  1962  and  consists  of 
eight  radio  configurations:  the  AN/VRC-12 

and  the  AN/VRC-43  through  AN/VRC-49--which 
provide  short-range  vehicular  communications 
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for  the  Army  in  the  field.  All  configura- 
tions consist  of  various  combinations  of  the 
three  major  components,  which  are  the  Receiver/ 
Transmitter  (RT-246  with  10  push  button  presets) , 
the  Receiver/  Transmitter  (RT-524  with  a 
built-in  loudspeaker) , and  the  Auxiliary 
Receiver  (R-442).  Common  modules  are  utilized 
for  all  three  major  components,  and  all 
configurations  may  be  readily  adapted  for  use 
in  any  tracked  or  wheeled  vehicle  or  they  can 
be  used  alone  or  in  conjunction  with  ancillary 
vehicular  radio  intercommunications  equip- 
ment, such  as  the  Intercommunications  Set, 

AN/VIC-1 (U).  All  AN/VRC-12  family  configur- 
ations use  the  AS-1729/VRC,  10-foot,  omni- 
directional vehicular  whip  antenna.  A more 
detailed  description  of  the  technical  charac- 
teristics of  this  vehicular  radio  is  provided 
in  Appendix  E. 

AN/ARC-114 . This  airborne  radio,  which  was 
operationally  deployed  in  1970,  is  a modularly 
constructed,  securable  VHF-FM  radio  receiving 
and  transmitting  set  used  for  Army  tactical 
aircraft  command  and  control  communications. 

It  is  interoperable  with  the  AN/PRC-77  and 
AN/VRC-12  families  of  radios  and  can  be  tuned 
to  any  one  of  920  channels.  In  addition,  it 
has  a fixed  tuned  guard  receiver.  The 
AN/ARC- 114A  provides  a homing  capability  when 
used  with  the  FM  Homing  Antenna  System  and 
Heading-Radio  Bearing  , Indicator,  ID-1351  1A. 

The  radio  set  has  no  organic  antenna  because 
the  antenna  is  integral  to  the  aircraft  and 
varies  with  the  type  of  aircraft.  A more 
detailed  description  of  this  airborne  radio 
is  provided  in  Appendix  E. 

• Alternative  2:  Product  Improvement  of  the  Current 

Series  of  CNRs.  As  the  term  product  improvement 
implies,  the  current  series  of  CNRs  will  simply  be 
improved  in  terms  of  reliability.  Complete  modification 
of  the  radios  to  meet  the  SINCGARS-V  Required 
Operational  Capability  (ROC)  characteristics  is 

not  contemplated.  Available  channels  will  increase 
to  1040 — spaced  at  25  KHz.  A more  detailed  descrip- 
tion of  the  technical  characteristics  of  the  product- 
improved  CNRs  is  provided  in  Appendix  E. 

• Alternative  3 : Developmental  Radio  Set,  AN/URC-78 . 

This  radio,  which  uses  state-of-the-art  technology  to 
include  Large  Scale  Integration  (LSI) , has  taken  a 
system  approach  to  the  Army  VHF-FM  tactical 
communications  requirements.  It  provides  modules 
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to  sat'sfy  the  various  manpack,  vehicular  and 
airborne  radio  requirements.  The  basic  module  is 
the  receiver/  transmitter  (R/T)  which  is  common  to 
all  configurations.  Appliques  and  adapters  in  the 
form  of  power  amplifiers,  antennas,  and  vehicular 
mounts  are  used  to  configure  the  radio  o meet 
specific  requirements.  Unique  features  of  the 
AN/URC-78  include: 

All  electronic  channel  selection 

Tour  preset  channels 

Automatic,  continuous  antenna  tuning 

A low  antenna  silhouette,  involving  a 1 to  4- 
foot  manpack  whip  and  a 6-foot  vehicular  whip 
antenna 

.electable  25  KHz/50  KHz  channel  spacing 

The  AN/URC-78  is  compatible  with  the  current 
families  of  VHF-FM  CNRs,  their  intercommunications 
systers,  radio/wire  integration  (R/WI)  systems, 
and  ot.ier  accessories.  The  AN/UFC-78  radio  system 
has  nc  inherent  electronic  counter-countermeasure 
(ECCM)  capabilities  other  than  operator-applied 
techniques,  such  as  alternate  frequency  selection. 
Additional  details  with  respect  to  the  technical 
characteristics  of  this  radio  are  provided  in 
Appendix  E. 

• Alternative  4:  A New  CNR.  A nev  CNR  will  involve 

the  application  of  the  latest  tec  hnology  to  provide 
the  best  technical  approach  ( BTA)  to  the  problem 

of  ensuring  that  the  Army  in  the  field  will  have 
the  most  suitable  VHF-FM  CNR  in  the  post-1980 
period  of  time.  Among  the  other  ' proved  features 
of  this  radio  will  be  an  ECCM  ca:  .lity  for  those 

manpack,  vehicular,  and  airborne  modules  which 
require  such  a capability.  A more  detailed 
description  of  the  technical  characteristics  of 
this  new  CNR  is  provided  in  Appendix  F. . 

The  primary  implication  of  the  foregoing  summary  of  the 
various  types  of  radio  systems  involved  in  the  four  alterna- 
tive courses  of  action  identified  ny  the  SINCGARS  Special 
Task  Force  is  that: 

• There  are,  in  fact,  different  types  of  radio  sets 
with  different  performance/physical  characteristics 
and  different  required  quantities  to  be  considered 
in  connection  with  Alternatives  1 and  2.  Judgments 
with  respect  to  the  utility  of  i he  various  perfor- 
mance/physical characteristics  of  thj  radio  sets 

in  Alternatives  1 and  2 therefore  reflect  corroosite 
systems.  Moreover,  it  should  be  noted  that  the 
overall  utility  values  for  each  of  the  various 
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alternatives  reflect  the  ombined  values  for  the 
three  different  types  of  platforms. 


2.0  PROJECT  OBJECTIVE 


The  objective  of  the  research  reflected  in  this  report 
was  to  develop  and  test  a multi-attribute  utility  model 
which  might  be  used  to  assist  the  SINCGARS  STF  in  evaluating 
the  military  utility  of  alternative  CNR  configurations.  Of 
particular  interest  in  the  development  of  this  model  was  the 
need  to  provide  an  analytic  tool  which  would: 

• Provide  a comprehensive  evaluation  of  the  four 
alternatives . 

• Be  able  to  accommodate  new  alternatives  as  well  as 
modifications  to  the  given  alternatives. 

• Accept  "last  minute"  changes  in  data  and  diverse 
expert  judgments. 

• Enable  a user  to  conduct  "what  if"  analyses  on  an 
iterative,  interactive  basis. 


3.0  GENERAL  STRUCTURE  OF  THE  MODEL 


3 . 1 Approach 

A variety  of  procedures  might  be  employed  for  the 
purpose  of  evaluating  alternative  systems  — in  this  case, 
alternative  radio  systems.  These  procedures  could  range 
from  the  elicitation  of  overall  (global)  judgments  with 
respect  to  the  alternative  systems  from  a group  of  experts 
to  the  use  of  very  complex  computer  simulations.  The  former 
approach  tends  to  be  too  simplistic  and  subject  to  many 
inherent  difficulties  — one  of  which  is  that  the  problem  is 
far  too  complex  for  accurate,  global  judgments  — and  the 
latter,  although  often  used,  suffers  from  the  shortcoming 
that  system  performance  may  not  be  directly  related  to 
measures  of  worth  or  utility  for  mission  performance. 

The  approach  adopted  during  this  project  was  to  develop, 
test,  and  utilize  on  an  iterative  basis  what  is  described  as 
a multi-attribute  utility  model,  which  decomposes  the  prob- 
lem in  such  a way  that  independent  judgments  may  be  made 
with  regard  to  individual  components  of  the  problem  and 
subsequently  aggregated  to  provide  an  overall  judgement 
using  a formal  mathematical  structure  which  can  be  imple- 
mented on  an  interactive  computer.  The  model  utilizes 
utilities  derived  from  user  requirements.  Such  utilities 
are  supplied  by  members  of  the  user  community.  The  model 
is  tied  to  the  technical  performance  characteristics  of  a 
system,  thus  emphasizing  how  a system  performs  as  opposed  to 
how  such  perfornance  is  achieved. 

The  advantages  of  such  a model  are  two-fold,  namely: 

• It  permits  ar.  individual  who  is  an  expert  in  a 
particular  aiea  to  make  judgments  which  involve 
his  particular  area  of  expertise  rather  than 
making  an  ovc  rail  judgment  of  worth  which  may  fall 
outside  his  erea  of  expertise. 

• Disaggregating  the  judgments  of  individual  experts 
provides  a clearly  auditable  trail  leading  from 
measures  of  system  performance  to  measures  of 
benefit  or  utility.  Thus  the  judgments  are  public 
rather  than  private  and  are  subject  to  screening. 

In  other  words, 

Being  able  to  separate  overall  judgments  into  (explicit) 
components  in  this  manner  can  provide  a manager  with 
valuable  information  about  the  relative  importance  of 
various  attributes  of  a product  (or  system) . It  can 
also  provide  a manager  with  valuable  information  about 
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lit  it  ki ij. 


4 he  value  of  various  levels  of  a sing.e  attribute, 
ndeed,  seme  models  can  even  estirrate  the  (implicit) 

Jrade-offs  consumers  (users)  2itiake  wher  they  evaluate 
everal  attributes  together. 

Thus,  although  a multi-attribute  utility  model  (such  as 
will  be  described  in  this  report)  is  dependent  upon  the  expert 
judgment  of  the  user  community,  it  is  objective  in  that  the 
linkages  between  data  and  conclusions  are  identified  and 
clearly  visible. 

At  the  very  outset,  it  is  important  to  differentiate 
between  the  multi-attribute  utility  model  which  has  been 
developed  and  other  forms  of  models  which  are,  in  fact, 
simulations  of  the  system  being  developed.  On  the  one  hand, 
the  multi-attribute  utility  model  shows  how  changes  in 
system  parameters,  which  describe  system  performance  and 
acceptability,  enhance  or  reduce  the  attractiveness  of  a 
proposed  system  for  the  user.  A simulation  model,  on  the 
o^  her  hand.-  generally  shows  how  changes  in  technological 
characterist j js  modify  the  technical  performance  of  the 
system. 

3 . 2 Expert  Judgments 

As  may  be  appreciated,  a crucial  step  in  developing  a 
multi-attribute  utility  model  involves  the  identification  of 
the  experts  whose  judgments  will  serve  as  inputs  during  the 
modeling  process.  The  development  of  the  structure  of  the 
STNCGARS  model  and  the  formulation  of  parameter  ranges, 
utility  functions,  rules  of  combination  and  importance 
weights  were  accomplished  during  a ser  Les  of  intensive 
working  sessions  with  SINCGARS  STF  personnel  representing 
the  user.  The  judgments  of  these  individuals  were  further 
refined  by  using  mission  and  performance  envelopes  provided 
by  the  Army  Training  and  Doctrine  Command  (TRADOC) . 

3.3  An  Overview  of  the  Structure  of  the  Model 


The  first  step  in  developing  the  SINCGARS  Evaluation 
Model  was  to  establish  a logical  structure  relating  physical 
system  parameters  to  military  value.  This  structure  is  a 
model  describing  how  the  physical  and  performance  parameters 
of  a radio  system  are  translated  into  measures  of  utility 
reflecting  how  well  that  system  accomplishes  its  military 
mission.  Although  a variety  of  structural  decompositions 
are  possible,  assessment  of  the  military  value  of  a system 
is  usually  decomposed  into  an  assessment  of  technical  system 
utility  (i.e.,  how  well  the  system  functions  as  a radio)  and 
an  assessment  of  operational  acceptability  (i.e.,  how  attractive 
the  system  i ; from  an  operational  point  of  view)  . The 
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assessments  of  these  two  dimensions  are  ultimately  combined 
to  provide  a measure  of  military  value  which  may  then  be 
related  to  a measure  of  the  cost  of  the  particular  system. 

In  developing  the  structure  of  this  multi-attribute 
utility  model,  the  dimensions  of  technical  system  utility 
and  operational  acceptability  are  further  decomposed  into 
sub-dimensions.  For  example,  technical  system  utility  may 
be  decomposed  into  technical  performance  (i.e.,  how  well  the 
system  performs)  and  dependability  (i.e.,  how  reliable/ 
available  the  system  is  considered  to  be) . Each  of  these 
dimensions  may  be  further  decomposed  so  that,  as  the  decom- 
position continues,  a hierarchy  consisting  of  several  levels 
of  sub-dimensions  or  factors  is  created  — each  level  becoming 
more  specific  than  the  previous  level.  Decomposition  of  the 
sub-dimensions  is  complete  when  the  sub-dimensions  may  be 
specified  or  quantified  by  relating  them  to  one  or  more 
technical  performance  cha  ■ act.eristics  c f the  system.  For 
example,  the  receive/tran:;mit  (R/T)  capability  of  a SINCGARS 
may  be  quantified  in  terms  of  the  communications  planning 
range  (CPR)  by  radio  platform  (i.e.,  manpack,  vehicular,  and 
airborne) . The  lowest  levels  of  the  mcdel  are  thererore 
actual  measurable  system  characteristics,  and  the  overall 
structure  of  the  model  serves  as  a mechanism  for  com)  ining 
the  information  obtained  with  regard  tc  these  paramte  rs  into 
measures  of  utility. 

The  general  structure  of  the  SINCGARS  Evaluation  Model 
which  was  developed,  highlighting  technical  system  utility, 
is  shown  in  Figure  3-1.  The  further  decomposition  of  tech-  i 

nical  performance  into  its  constituent  sub-dimensions  is 
reflected  in  Figure  3-2.  A similar  diagram  of  the  structure 
of  the  SINCGARS  Evaluation  Model,  highlighting  operational 
acceptability,  is  presented  in  Figure  3-3.  As  ma*  be  observ^ 
in  this  figure,  the  physical  characteristics  dimension  of 
operational  acc  jptability  has  been  further  decomposed  into 
such  sub-dimensions  as  weight,  size,  form  factor,  visual 
, detectability,  human  factors  engineering,  and  security 

features.  Moreover,  the  sub-dimension  pertaining  to  size 
has  been  quantified  in  terms  of  the  volume  (cubic  inches)  of 
the  radio  by  platform  (i.e.,  manpack,  vehicular,  and  air- 
borne) . The  decomposition  of  the  other  dimensions  of  opera- 
i tional  acceptability  (viz.,  flexibility,  survivability, 

i support  requirements  and  transmission  quality)  is  reflected 

j in  Figures  3-4,  3-5,  3-6  and  3-7. 

\ 3.4  Condition: ng  Variables  of  the  Model 

Between  the  level  of  the  major  dimensions  of  value  and 
that  of  the  sys  ;em  parameters/sub-paran eters  in  Figures  3-1 
and  3-3  are  thr  ?e  levels  identified  as  scenarios,  natural  j 

environments,  fid  platforms.  The  variables  described  at  j 

I each  of  these  .i  itervening  levels  are  not  sub-dimensions  of  1 

i the  major  dimer  lions  of  value,  ncr  are  they  system  parameters/ 

sub-parameters.  Instead,  they  are  conditioning  variables  . 

i 
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upon  which,  for  any  SINCGARS  alternative,  the  values  of  the 
system  parameters/sub-parameuers  and  their  associated  utilities 
will  depend.  In  other  words,  the  performance  and  utility  of  a 
SINCGAP.S  alternative  will  vary  as  a function  of  the  situation 
or  scenario  in  which  it  is  employed,  the  natural  environments 
in  which  it  will  perform,  and  the  platform  on  which  it  is 
mounted.  In  this  model,  the  scenarios  have  been  broadly 
defined  as  follows: 

• Forward  edge  of  the  battle  area  (FEBA)  oriented 
operations,  in  which  '-.here  is  a generally  defined, 
continuous  front  which  moves  forward  or  backward 
as  a battle/campaign  progresses. 

• Non-FEBA  oriented  operations,  in  which  the  tactical 
situation  is  quite  fluid  and  the  front  is  generally 
non-continuous . 

The  second  conditioning  variable,  v/hich  reflects  the  various 
environments,  has  been  broadly  defined  as  follows: 

• Forested  areas,  such  as  might  be  illustrated  by 
the  Bavarian  area  of  southern  Germany  in  which 
communications,  ground  mobility,  and  visual  ob- 
servation are  generally  inhibited  by  the  heavy 
growth  of  trees  and  underbrush. 

• Mountainous  areas,  such  as  might  be  represented  by 
the  precipitous  terrain  in  eastern/central  Korea  and 
parts  of  the  Middle  East,  in  which  communications, 
ground  mobility,  and  visual  observation  are  generally 
inhibited  by  intervening  land  masses. 

• Urban  areas,  such  as  those  which  cover  a con- 
siderable portion  of  the  landscape  of  central 
Germany  and  generally  inhibit  communications, 
ground  mobility,  and  visual  observation. 

• Open  country,  such  as  the  central  plains  of  Germany 
and  France  and  parts  of  the  Middle  East  where 
there  is  little  degradation  of  communications  and 
visual  observation  due  to  terrein  and  ground 
mobility  is  generally  excellent: . 

The  third  and  final  important  conditioning  variable,  as 
a function  of  which  the  performance  and  utility  of  a radio 
system  will  vary,  is  the  platform  on  which  the  radio  will 
operate.  The  three  categories  cf  platforms  are  generally 
described  as  follows: 

• Manpack,  which  indicates  that  the  normal  means  of 
transporting/operating  the  radio  is  the  individual 
soldier  (s) . 


15 


• Vehicular,  which  indicates  that  the  radio  is 
normally  mounted  in/operated  from  u wheeled  or  a 
tracked  vehicle. 

• Airborne,  which  indicates  that  the  radio  is  nor- 
mally mounted  in/operated  from  an  Army  helicopter 
or  fixed  wing  aircraft. 

As  may  be  readily  observed  in  Figures  3-1  and  3-3,  the 
two  scenarios,  four  environments,  and  three  categories  of 
platforms  provide  a total  of  24  (i.e.,  2x4x3)  different 
paths  or  single  threads  from  the  level  of  the  major  dimen- 
sions of  value  to  that  of  the  system  parameters.  The  values 
of  the  system  parameters  are  determined  for  each  of  the  24 
threads  and  then  the  utility  over  each  parameter  and  sub- 
parameter is  assessed,  conditional  upon  the  particular 
thread  that  is  being  evaluated.  Ultimately,  an  overall 
utility  for  a given  system  may  be  obtained  by  combining  the 
utilities  associated  with  each  of  the  24  threads. 

3. 5 Specification  of  the  Sub-Dimensions  of  the  Model 

As  previously  indicated  on  pages  7 and  8,  the  sub- 
dimensions at  the  lowest  levels  shown  in  Figures  3-1  and  3-3 
are  specified  in  terms  of  one  or  more  system  parameters. 

For  example, 

• The  receive/transmit  (R/T)  capability  of  a SINCGARS 
is  specified,  as  reflected  in  Figure  3-2,  in  terms 
of  the  communications  planning  range  (CPR)  by 
platform. 

• The  channel  capabilities  of  a SINCGARS  are  spe- 
cified in  terms  of  the: 

Total  number  of  channels  available 
Number  of  preset  channels  available 
Tuning  capabilities 

In  other  cases,  the  sub-dimensions  are  specified  by  the  res- 
ponses to  one  or  more  questions.  For  example, 

• Burst  transmission  capability  under  technical 
flexibility  in  Figure  3-2  is  described  by  a "yes" 
(indicating  that  the  radio  has  the  capability  and 
is  therefore  given  a value  of  1.0)  or  a "no" 
(indicating  that  the  radio  does  not  have  the 
capability  and  therefore  is  given  a value  of  0.0). 

• Interoperability  under  operational  flexibility 
in  Figure  3-4  is  described  on  the  basis  of  the 
degree  of  interoperability  of  a SINCGARS  alter- 
native with  other  related  radios.  There  are  three 
degrees  of  interoperability. 
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In  developing  the  SINCGARS  Evaluation  Model,  it  was 
necessary  to  assign  weights  to  each  of  the  factors  which 
entered  into  the  model  and  to  generate  utility  functions  for 
the  controllable  system  performance  characteristics.  How- 
ever, before  doing  this,  it  was  necessary  to  establish 
reasonable  ranges  for  each  of  the  system  performance  charac- 
teristics because  the  interval  over  which  the  technical 
performance  characteristics  can  vary  will  affect  both  the 
shape  of  each  utility  function  and  the  magnitude  of  the 
weights  assigned  to  particular  factors.  For  example,  al- 
though a given  parameter  might  be  extremely  important  in 
principle,  its  importance  may  not  change  with  respect  to  the 
range  under  consideration,  and  it  will  therefore  receive  a 
lesser  weight  than  one  whose  range  varies  significantly  for 
the  alternative  systems  under  consideration.  The  retrans- 
mission capability  of  a radio  system  is  undoubtedly  an 
important  function,  but  if  all  of  the  SINCGARS  alternatives 
under  consideration  have  equivalent  retransmission  capa- 
bilities, then  this  sub-dimension  should  receive  little  or 
no  weight  in  the  model. 

3 . 6 Utility  Functions 

Once  the  lowest  level  sub-dimensions  of  value  have  been 
described  in  terms  of  system  performance  characteristics  and 
reasonable  ranges  have  been  established  for  each  of  these 
characteristics,  then  a utility  function  is  assessed  over 
the  range  for  each  of  the  system  performance  characteristics. 
This  function  assigns  a utility  of  1.0  to  the  best  va’ue  in 
the  range  of  the  performance  characteristic  and  0.0  to  the 
worst  value  in  the  range  of  the  characteristic.  All  inter- 
mediate values  are  assigned  utilities  between  0.0  and  1.0. 

The  utility  function,  which  may  be  cont  nuous  (as  shown  in 
Figure  3-8)  or  may  be  discrete  (as  shown  in  Figure  3-9) , 
serves  to  translate  changes  in  the  system  characteristics 
into  a measure  of  benefit  conditioned  by  the  particular  path 
through  the  hierarchy  with  which  a particular  utility  func- 
tion is  associated.  The  utility  functions  also  provide  a 
means  of  assessing  the  impact  on  the  overall  system  ptrfor- 
. ince  of  marginal  changes  in  a number  of  technical  system 
characteristics . 

The  manner  in  which  a utility  function  captures  the 
importance  of  a variation  in  performance  characteristics  is 
illustrated  by  means  of  the  curves  shown  in  Figures  3-10  and 
3-11.  In  the  case  of  kit  installation  time  in  Figure  3-10, 
an  installation  time  of  zero  is,  of  course,  ideal,  but  an 
installation  time  of  20  minutes  still  has  a utility  oi  0.6 
on  a scale  from  0 to  1.0.  On  the  other  hand,  in  the  case  of 
transmission  quality  in  Figure  3-11,  a transmission  quality^ 
for  data  greater  than  10  bits  per  second  is  good,  but  10  * 
bits  per  second  is  useless.  Therefore,  the  latter  utility 
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Figure  3-8 

BASIC  UTILITY  FUNCTION  FOR  THE  TOTAL  NUMBER 
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Figure  3-9 

BASIC  UTILITY  FUNCTION  FOR  NUMBER  OF 
ADDITIONAL  SIMULTANEOUSLY  MONITORED  CHANNELS1 

Without  the  use  of  an  auxiliary  reccivei 
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function  essentially  represents  a point  requirement  for 
transmission  quality.  The  advantages  of  using  utility 
functions  as  opposed  to  point  requirements  for  the  evalua- 
tion of  alternative  systems  are  that  utility  functions: 

• Preclude  the  over-design  of  a system  by  explicitly 
acknowledging  the  fact  that  the  design  process  is 
a constant  trade-off  between  benefits  received,  on 
the  one  hand,  and  the  cost  of  achieving  those 
benefits,  on  the  other  hand. 

• Permit  identification  of  those  areas  in  which 
requirements  may  be  relaxed  with  little  loss  in 
total  system  utility  versus  those  areas  in  which  a 
small  change  in  requirements  may  have  a major 
impact  on  total  system  utility. 

In  all,  there  were  some  94  utility  functions  developed 
for  all  of  the  system  parameters  used  to  describe  the  sub- 
dimensions of  value  in  the  SINCGARS  Evaluation  Model.  In 
addition,  there  were  almost  1300  pieces  of  input  data  required 
for  the  various  system  parameters/sub-parameters  of  the  alter- 
native systems  evaluated. 

3 . 7 Rules  of  Combination 

The  purpose  of  the  hierarchical  utility  structure  shown 
in  Figures  3-1  and  3-3  is  to  transform  technical  descrip- 
tions of  system  performance  at  the  bottom  of  the  structure 
into  a measure  of  military  value  at  the  top  of  the  structure. 
The  rules  for  aggregating  factors  within  each  level  of  the 
hierarchy  are  either  additive  or  multiplicative,  depending 
upon  whether  or  not  the  factors  contribute  to  overall  mili- 
tary value  in  an  independent  or  interactive  manner.  The 
interactions  that  are  important  in  a utility  model  are 
value-wise  dependencies — not  technical  system  dependencies. 

In  other  words,  variables  can  be  independent  in  terms  of 
their  contribution  to  overall  system  utility  but,  due  to 
design  constraints,  they  interact  strongly  in  terms  of  what 
might  be  realized  with  respect  to  a particular  system. 

At  the  scenario,  environment,  and  platform  levels  of 
the  model,  the  variables  are  essentially  independent,  and 
the  overall  utility  of  an  alternative  system  increases  with 
the  utility  of  each  variable.  The  variables  are  therefore 
combined  additively.  At  the  dimension/sub-dimension  levels 
of  the  model,  the  factors  are  combined  additively  and 
multiplicatively . For  example,  as  indicated  in  Figure  3-1, 
the  sub-dimensions  of  technical  performance  are  all  additive, 
but  the  sub-dimensions  of  dependability  are  multiplicative. 
Furthermore,  as  also  may  be  observed  in  Figure  3-1,  the 
technical  performance  and  dependability  dimensions  are 
combined  multiplicatively  because  even  a radio  set  with 
outstanding  technical  performance  capabilities  could  be 
degraded  to  zero  if  it  is  undependable. 


The  actual  rules  of  combination  applied  throughout  the 
SINCGARS  Evaluation  Model  are  either  + (additive)  or  x 
(multiplicative),  as  shown  in  Figures  3-1  through  3-7. 

3.8  Importance  Weiqhts 

After  ranges  are  established  for  each  technical  para- 
meter/sub-parameter and  utility  functions  are  developed,  it 
is  necessary  to  assign  importance  weights  to  each  branch  of 
the  model.  Two  kinds  of  importance  weights  are  required, 
depending  upon  the  rules  of  combination.  In  the  case  of  a 
set  of  additive  factors,  importance  weights  simply  provide  a 
means  of  calculating  a weighted  average  over  the  factors 
being  considered.  Consequently,  the  importance  weights 
assume  the  form  of  coefficients  that  are  multiplied  by  the 
utility  of  the  system  for  each  of  the  factors,  and  these 
coefficient's  sum  to  1.0  across  all  factors  within  a given 
level  of  the  hierarchical  structure.  The  weight  given  to  a 
particular  factor  reflects  the  relative  contribution  of  that 
factor  to  utility  for  that  level  of  the  model. 

In  the  event  that  factors  within  a given  level  are 
multiplicative  (that  is,  they  are  interactive  in  a value- 
wise  sense) , then  the  measure  of  utility  of  any  factor  may 
be  considered  as  a measure  of  degradation.  When  variables 
interact  in  this  manner,  importance  weights  have  the  effect 
of  re-scaling  the  factors.  Prior  to  applying  a weight,  the 
utility  over  one  of  the  factors  may  have  any  value  between 
zero  and  one.  However,  assume,  for  example,  when  this  factor 
takes  on  its  worst  value,  then  the  effectiveness  of  a 
system  is  degraded  to  the  order  of  30%.  In  this  case,  the 
utility,  originally  scaled  from  0 to  1.0,  would  be  rescaled 
from  0.3  to  1.0. 

As  previously  indicated,  the  technical  performance  and 
dependability  dimensions  in  the  model  are  combined  multipli- 
catively  because  an  undependable,  though  highly  operable, 
radio  is  quite  useless.  In  the  case  of  this  model,  the 
dependability  factor  continues  to  be  scaled  from  0 to  1.0 
because  the  military  value  of  a thoroughly  undependable 
radio  is  considered  to  be  zero. 

3.9  Computer  Implementation  and  Evaluations 


Having  structured  the  model  as  described  in  the  pre- 
vious sub- sections , it  was  programmed  on  a computer  so  that 
it  could  be  utilized  on  an  interactive,  iterative  basis. 
Although  a model  such  as  this  may  be  substantively  logical 
to  those  involved  in  this  development,  the  initial  results 
from  the  model,  when  first  applied  to  the  evaluation  of 
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actual  systems,  may  identify  areas  where  modification  is 
required.  Therefore,  computer  implemeitatior  provides  not 
only  a capability  to  change  the  inputs  to  the  model  and 
rapidly  obtain  new  results  on  an  interative  basis,  but  also 
a capability  to  coi duct  extensive  sensitivity  analyses  both 
within  the  components  of  the  model  and  with  respect  to  the 
evaluation  of  diff  rent  kinds  of  systems  in  order  to  iden- 
tify those  portion;  of  the  model  which  requir 3 modification. 

Sensitivity  analyses  within  the  model  involve  an  evalua- 
tion of  the  degree  to  which  changes  in  importance  weights, 
combination  rules,  and  utility  functions  influence  the 
output  of  the  model.  For  example,  to  what  degree  does  the 
relative  utility  of  Alternative  System  1 versus  Alternative 
System  2 depend  upon  the  weights  assigned  to  a factor  within 
the  model?  Sensitivity  analyses  external  to  the  model 
provide  a means  of  calibrating  the  model  and  involve  the  use 
of  the  model  to  compare  known,  well-understood  alternative 
systems  before  using  the  model  to  evaluate  less  well  under- 
stood systems.  This  is  to  say  that  it  is  useful  to  test 
the  model  against  educated  judgments  with  respect  to  the 
relative  utility  of  existing  systems,  or  components  of 
existing  system  , before  applying  the  model  to  possible 
future  systems.  If  the  model  appears  to  be  blatantly  at 
odds  with  expert  judgments  pertaining  to  the  relative  utility 
of  known  systems,  analysis  of  the  model  should  lead  to  the 
identification  of  the  source  of  the  disagreement  and  thereby 
facilitate  modification  of  the  model  so  as  to  bring  subse- 
quent evaluations  more  into  line  with  informed  judgments. 

It  is  difficult  to  overemphasize  the  importance  of  these 
kinds  of  sensitivity  analyses  in  terms  of  both  improving  the 
evaluation  model  itself  and  developing  confidence  and  under- 
standing on  the  part  of  the  user  who  intends  to  make  recom- 
mendations based  upon  the  output  of  the  model. 

Several  sensitivity  analyses  were?  performed  on  the 
SINCGARS  Evaluation  Model.  Internal  sensitivity  analyses 
examined  the  sensitivity  of  the  model  with  respect  to  varia- 
tions in  importance  weights.  In  this  context,  it  has  been 
found  that  experts  consider  the  assessment  of  inter-dimen- 
sional importance  weights  to  be  more  difficult  than  judgments 
with  respect  to  inura-dimensional  utility  functions.  This 
is  not  surprising  since  the  inter-dimensional  importance 
weights  usually  involve  trading  off  quantities  measured  in 
different  units,  while  intra-dimensional  utility  functions 
only  involve  trading  off  different  amounts  of  a given  dimension. 
Errors  of  judgment  are  therefore  more  likely  to  occur  with 
respect  to  inter-d- mensional  importance  weights.  It  would  appear 
to  be  a reasonable  assumption  that,  even  if  an  expert  has  great 
difficulty  in  assessing  weights  over  factors,  the  weights 
assigned  should  reflect  the  correct  ordering  of  the  factors. 

The  limiting  case  of  error  and/or  variation  in  weights 
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occurs  when  all  weights  are  set  equal  — in  which  case  the 
order  of  factors  collapses,  for  they  are  all  equally  impor- 
tant. Hence,  the  equal  weighting  of  factors  does  constitute 
a good  internal  sensitivity  analysis  with  respect  to  this 
source  of  error. 

Table  3-1  presents  the  overall  values  of  technical 
systems  utility  and  operational  acceptability  which  were 
generated  by  the  model  for  alternative  radio  systems  iden- 
tified by  the  SINCGARS  Special  Task  Force. 


Table  3-1 

TECHNICAL  SYSTEM  UTILITY  AND  OPERATIONAL 
ACCEPTABILITY  WITH  UNEQUAL  AND  EQUAL  WEIGHTS 


Alternative  , 
Radio  Systema1 

Technical  System  Utility 

Operational  Acceptability 

Unequal  Weights 

Equal  Weights 

Unequal  Weights 

Equal  Weiqhtt 

KWS 

.03 

.03 

.32 

.35 

i 

. 34 

.30 

.69 

.60 

2 

. 39 

.34 

.73 

.65 

3 

.67 

.62 

.74 

.66 

4 

.65 

.59 

.80 

.68 

1The  various  radio  syrhems  may  be  briefly  identified  as  follows- 
KWS  - An  older  system,  vintage  of  the  Korean  War  ( 1 *1  "5 0 s 1 . 

1 - The  current  system,  vintage  of  the  196fs. 

2 - A product-improved  version  of  the  current  system. 

3 - A dovelonmental  svster.  with  a 4-foot  antenna. 

4 - A conceptual  system. 
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••  The  values  which  appear  in  Table  3-1  reflect  the  use  of 

both  unequal  and  equal  weights  at  all  nodes  of  the  structure 
of  the  model.  Although  the  actual  values  of  the  utilities 
i * of  the  alternative  systems  changed,  the  ordering  remained 

the  same.  The  values  of  operational  acceptability  are 
nearly  equal  for  Systems  2,  3,  and  4 with  equal  weights. 

This  is  because  the  systems  do  not  differ  greatly  with 
; respect  to  survivability,  support  requirements,  and  trans-  '■ 

i ; mission  quality.  The  main  differences  occur  in  terms  of  j 

i flexibility  and  physical  characteristics,  especially  the  • 

; latter,  and  these  two  dimensions  are  heavily  weighted  in  the 

j i model.  Assigning  equal  weights  then  places  relatively  less  j 

j emphasis  on  them  and  more  on  dimensions  where  the  differences  s 

are  extremely  small.  This,  of  course,  should  and  did  drive  i 

• l the  values  toward  equality.  Even  in  this  case,  however,  the  5 

f ordering  of  the  alternatives  remained  invariant.  Thus  it  j 

I may  be  stated  that  the  conclusions  reached  are  insensitive  j 

f * to  a fairly  broad  range  of  errors  in  weights.  This  is 

!•  consistent  with  results  obtained  in  other  decision  analytic  j 

studies.  i 

* I 

{ 

i For  example,  "Unit  Weighting  Schemes  for  Decision  Making,"  I: 

j by  H.  J.  Einhorn  and  R.  M.  Hogarth  in  Organizational  J; 

i Behavior  and  Human  Performances,  pages  13  and  l‘)l-192,  1975.  L 


U: 
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Such  insensitivity  is  not  exhibited,  however,  with 
respect  to  extreme  differences.  For  example,  assigning  some 
weights  of  0.0  and  others  of  1.0  has  the  effect  of  essentially 
deleting  certain  factors  and  highlighting  others.  Assign- 
ment of  such  weights  will  produce  substantial  changes  in  the 
ordering  (as  it  should) , for  the  model  is  highly  sensitive 
to  the  factors  included  therein. 

Insensitivity  of  a model  to  errors  in  weights  is 
desirable.  Insensitivity  to  differences  among  alternative 
systems  in  terms  of  several  system  parameters  is  undesirable. 
As  is  illustrated  in  Table  3-1,  the  SINCGARS  Evaluation 
Model  is  sensitive  to  differences  amon<r  alternative  systems 
and  permits  identification  of  the  reasons  for  the  differences 
in  utility.  Table  3-1  also  reflects  another  sensitivity 
analysis  external  to  the  SINCGARS  Evaluation  Model  which 
involves  the  establishment  of  benchmarks  that  permit  the 
interpretation  of  differences  in  utilities.  In  this  case, 
an  older  family  of  radios  (the  AN/GRC  3-8,  AN/PRC  9-11,  and 
AN/ARC-44)  of  the  Korean  War  era  was  evaluated.  The  utility 
values  obtained  for  this  series  of  radios  provide  a benchmark 
with  which  the  utilities  of  the  other  alternative  radio 
systems  may  be  compared.  This  comparison  provides  a basis 
for  such  questions  as: 

How  does  the  improvement  of  this  SINCGARS  alternative 
over  the  current  system  compare  with  the  improvement  of 
the  current  system  over  the  Korean  War  system? 

Tables  3-2  and  3-3  present  summaries  of  additional 
sensitivity  analyses  conducted  with  the  SINCGARS  Evaluation 
Model  in  which  selected  system  parameters  were  varied  through- 
out their  particular  ranges  of  values.  These  analyses 
permit  the  examination  of  the  impact  of  the  various  para- 
meters upon  overall  utility.  For  example,  in  Table  3-2,  it 
may  be  observed  that  airborne  communications  planning  range 
(ABNCPR)  has  a fairly  minimal  effect  upon  utility,  while  the 
retransmission  capability  (RETRAN)  has  a much  greater  effect. 
The  results  of  an  equal  weights  analysis  of  technical  system 
utility  are  also  presented  in  Table  3-2  and,  as  may  be 
observed,  the  relative  order  of  utilities  does  not  change. 

The  general  conclusion  that  was  reached  from  the  fore- 
going sensitivity  analyses  was  that  the  SINCGARS  Evaluation 
Model  does  perform  as  it  should.  It  is  fairly  robust  with 
respect  to  possible  errors  in  assessing  internal  utilities 
and  weights  and,  at  the  same  time,  it  is  externally  sensi- 
tive, i.e.,  sensitive  to  differences  in  alternative  systems. 
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Aircraft  CPR  (AIRCPR)  • Interface  (INTERF) 

Total  Channels  (TOTALN)  • Surat  Transmission  (BUR5TT) 


Table  3-3 

A SUMMARY  OF  SENSITIVITY  ANALYSES  FOR  THE 
OPERATIONAL  ACCEPTABILITY  PORTION  OF  THE  SINCGARS  EVALUATION  MODEL1 
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4.0  AN  EVALUATION  OF  HARDWARE  COST  ESTIMATES 


4.1  An  Overview 


In  order  to  relate  the  military  value  or  utility  of  the 
various  alternative  configurations  of  SINCGARS  with  the 
estimated  costs  of  these  alternatives,  a 15-year  life 
cycle  cost  (LCC)  estimate  was  generated  for  each  alternative 
utilizing  an  LCC  model  developed  by  the  Army  Electronics 
Command  (ECOM) . It  was  readily  recognized,  however,  that 
whereas  the  estimated  costs  of  Alternative  1 were  based  upon 
a wealth  of  experience  in  procuring  those  radios  to  date, 
the  cost  estimates  of  Alternatives  2,  3,  and  4 involve 
considerable  uncertainty.  Moreover,  in  examining  the  various 
cost  categories  in  the  15-year  LCC  model,  it  became  apparent 
that  the  major  cost  element  in  the  determination  of  the  LCC 
of  the  alternative  radio  configurations  is  the  average  unit 
cost  ($AUC)  of  the  radios;  i.e.,  the  Hardware  Cost  Element 
of  Recurring  Investment  which  constitutes  approximately  65% 
of  the  total  program  cost  over  15  years.  As  a consequence 
and  in  the  interests  of  creating  greater  confidence  in  just 
what  the  ultimate  costs  of  each  system  might  be,  an  inter- 
active Monte  Carlo  program  was  developed. 1 The  purpose  of 
this  program  was  to  combine  the  probability  distributions 
for  the  hardware  $AUC  of  each  radio  for  each  platform  (man- 
pack,  vehicular,  and  airborne)  in  proportion  to  the  Authorized 
Acquisition  Objectives  (AAOs)  of  those  radios/components. 

The  outputs  of  this  Monte  Carlo  program  were; 

• A probability  distribution  for  the  $AUC  of  each 
alternative  radio  system  by  platform.  An  illus- 
trative distribution  appears  in  Figure  4-1. 

• A probability  distribution  for  the  $AUC  of  each  of 
the  radios  and  their  ancillary  components  by 
platform.  An  illustrative  distribution  appears  in 
Figure  4-2. 

A summary  of  the  outputs  of  this  program  reflecting  not 
only  the  most  likely  $AUCs,  but  also  the  various  combinations 
of  radios,  ancillary  components,  and  AAOs  is  presented  in 
Tables  4-1,  4-2,  and  4-3. 

4 . 2  Procedures  and  0 ltput 

Probability  distributions  for  the  average  unit  cost 
estimates  associated  with  the  various  major  components  of 
the  alternative  SINCGARS  were  developed  on  the  basis  of 
elicitations  from  experts  selected  by  the  SINCGARS  STF . The 

^The  Monte  Carlo  Program  was  developed  under  a contract  sponsored 
jointly  by  the  Defense  Advanced  Research  Projects  Agency  (ARPA) 
and  the  Rome  Air  Development  Center  (RADC)  and  is  described  in 
a technical  report  entitled,  Graphic  Interactive  System  for 
Decision  Theoretic  Analysis  by  Decisions  and  Designs,  incorporated, 
December  3fl,  1974 . *" 
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Table  4 1 

A SUMMARY  OF  THE  MOST  LIKELY  AVERAGE  UNIT  COSTS  <$AUC}  FOR  ALTERNATIVE  2 


—Authorized  Acquisition  Objective 


Table  4-2 

SUMMARY  OF  THE  MOST  LIKELY  AVERAGE  UNIT  COSTS  ($AUC)  FOR  ALTERNATIVE  3 


Authorized  Acquisition  Objective 


Table  4-3 

A SUMMARY  OF  THE  MOST  LIKELY  AVERAGE  UNIT  COSTS  ($AUC)  FOR  ALTERNATIVE  4 


Authorized  Acquisition  Objective 


elicitation  process  was  facilitated  through  the  use  of  an 
interactive  computer  program  with  a graphic  display.  For 
each  of  the  major  components  of  Alternative  Systems  2,  3, 
and  4,  the  experts  were  initially  requested  to: 

• Define  the  minimum  feasible  and  maximum  feasible 
costs . 

• Trisect  the  range  of  costs  between  the  minimum  and 
maximum  feasible  so  as  to  provide  three  intervals — 
intervals  such  that  the  probability  that  the  cost 
of  the  particular  component  will  fall  in  each 
interval  would  be  the  same,  i.e.,  0.333. 

• Quadrisect  the  range  of  costs  between  the  minimum 
and  maximum  feasible  so  as  to  provide  four  intervals 
intervals  such  that  the  probability  that  the  cost 

of  the  particular  component  will  fall  in  each 
interval  would  be  the  same;  i.e.,  0.25. 

• Define  a 0.95  credible  interval — an  interval  such 
that  the  probability  that  the  cost  of  the  particular 
component  will  fall  in  this  interval  would  be 

0.95. 

The  computer  then  displayed  three  probability  distribu- 
tions reflecting  the  foregoing  inputs  by  the  experts. 
Inconsistencies  in  these  distributions  were  eliminated  by 
virtue  of  direct  interaction  between  the  computer  and  the 
experts  involving  iterative  adjustments  of  the  distributions. 
As  a result,  a consensus  was  attained  among  the  experts  with 
regard  to  the  probability  distribution  for  the  $AUC  of  each 
radio  and  major  component.  In  order  to  combine  these  $AUCs, 
the  Monte  Carlo  program  sampled  each  of  the  probability 
distributions  for  the  $AUC  in  accordance  with  the  various 
comninations  of  radios/components  and  quantities  (AAOs) 
shown  in  Tables  4-1,  4-2,  and  4-3.  A summary  of  the  most 
likely  $AUC  for  the  three  alternative  systems  by  platform  is 
provided  in  the  table  which  follows: 


Table  4-4 

A SUMMARY  OF  THE  MOST  LIKELY  SAUC  BY  ALTERNATIVE  AND  PLATFORM 


Platform 

$AUC  of 

MosJ  Likely 
Alternative  Systems 

2 

3 

4 

Manpack 

$1090 

$1590 

$1550 

Vehicular 

3190 

2980 

2570 

Airborne 

3980 

1990 

1970 

33 


The  foregoing  $AUCs  were  simply  a.'  rected  from  Tables 
4-1,  4-?,  and  4-3  and  reflect  the  mix  of  radios  and  major 
components  shown  in  those  tables. 

4.3  Observations 


Still  referring  to  Tables  4-1  through  4-4,  it  is  apparent 
that  Alternative  2 is  the  most  costly  system  and  Alternative 
4 is  the  least  costly  system.  Moreover,  it  may  be  observed 
that  the  primary  difference  between  the  $AUCs  of  Alternatives 
3 and  4 is  that  of  the  vehicular  radio.  This  difference  may 
be  attributed  to  the  fact  that  the  AN/VRC-12  and  AN/VRC-47 
combination  for  Alternative  4 employs  an  auxiliary  receiver, 
whereas  the  corresponding  combination  for  Alternative  3 uses 
two,  rather  than  one,  receiver/transmitter s (R/T) . The  $AUC 
of  this  R/T  is  nearly  $500  more  than  the  auxiliary  receiver 
of  Alternative  4. 

As  may  be  noted  in  examining  the  probability  distri- 
bution curves  for  $AUC,  the  curves  do  have  different  shapes. 
Hence,  the  modal  value  is  a gross  approximation  which  should 
not  be  used  in  isolation.  For  example,  the  most  likely  $AUC 
for  the  aircraft  radio  in  Alternative  4 is  less  than  the 
$AUC  for  the  aircraft  radio  in  Alternative  3.  However,  the 
resultant  probability  distribution  for  the  $AUC  of  the 
aircraft  radio  in  Alternative  3 is  much  more  symmetrical 
than  that  of  the  aircraft  radio  in  Alternative  4,  which 
reflects  a much  higher  probability  of  cost  overrun. 

As  a cross-check  on  the  assessment  of  the  overall  $AUC 
of  the  R/T  for  Alternative  3,  probability  distributions  were 
assessed  for  the  $AUC  of  each  of  the  nine  components  that 
constitute  the  R/T  of  Alternative  3.  The  resulting  distri- 
butions were  combined  using  Monte  Carlo  procedures  to  obtain 
an  overall  distribution  for  the  R/T  $AUC  of  Alternative  3, 
which  is  shown  in  Figure  4-3. 

Using  this  procedure,  the  most  likely  $AUC  for  the  R/T 
of  Alternative  3 is  $1455,  which  corresponds  rather  closely 
with  the  R/T  $AUC  of  $1410  in  Table  4-2.  However,  as  may  be 
observed  in  Figure  4-3,  the  probability  distribution  for  the 
R/T  $AUC  when  the  R/T  is  decomposed  into  its  major  components 
is  much  tighter — ranging  from  $1280  to  $1730.  The  overall 
probability  distribution  for  the  R/T  $AUC  reflected  in 
Figure  4-1  ranges  from  $1100  to  $2200  and  is  also  less 
symmetrical.  By  way  of  explanation,  the  usual  result  of 
combining  the  probability  distributions  for  the  $AUCs  of 
many  components  of  a system  is  that  the  resultant  proba- 
bility distribution  is  narrower  than  that  obtained  by 
assessing  the  probability  distribution  for  the  $AUC  of  the 
system  as  a whole.  The  reason  for  this  is  that  an  expert 
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Figure  4-3 

COMBINED  PROBABILITY  DISTRIBUTION  FOR  THE  $AUC  OF 
ALL  COMPONENTS  OF  THE  RECEIVER/TRANSMITTER  OF 
ALTERNATIVE  3 
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will  tend  to  be  much  more  certain  about  the  $AUC  of  each 
component  of  a system  than  he  will  be  with  respect  to  the 
$AUC  of  the  system  as  a whole. 


In  this  context,  it  is  safe  to  assume  that  a similar 
tightening  of  probability  distributions  would  occur  if  the 
R/T  $AUCs  for  Alternatives  2 and  4 were  decomposed  into 
component  $AUCs.  Once  again,  the  most  likely  $AUCs  would 
tend  to  agree,  but  the  decomposed  probability  distributions 
would  have  smaller  ranges.  Relying  upon  the  broader  overall 
probability  distribution  for  the  $AUC  of  a system  (rather 
than  the  component  probablity  distributions) , an  expert  may 
be  quite  certain  that  he  has  captured  the  relevant  range  of 
cost  which,  in  essence,  is  a more  risk-averse  approach.  In 
other  words,  when  actual  costs  are  assessed,  the  expert  is 
less  likely  to  be  surprised. 
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5.0  RESULTS  AND  CONCLUSIONS 


5 < 


Having  structured  and  tested  the  SINCGARS  Evaluation 
Model  as  described  in  Section  3.0  of  this  report,  the  four 
alternative  systems,  as  well  as  the  Korean  War  system,  were 
evaluated.  The  results  of  this  evaluation  appear  in  Table 
5-1.  In  Table  5-2,  not  only  are  the  values  of  technical 
system  utility  and  operational  acceptability  shown  for  each 
of  the  alternatives,  but  also  the  values  of  the  various  sub- 
dimensions of  these  major  dimensions.  These  utility  values 
have  been  aggregated  across  all  of  the  scenarios,  environments, 
and  platforms.  In  Table  5-3,  the  values  of  technical,  system 
utility  and  operational  acceptability  are  presented  for  each 
of  the  alternatives  as  a function  of  aggregating  the  platforms 
across  the  scenarios  and  environments. 


A complete  tabulation  of  the  utility  values  for  technical 
system  utility  and  operational  acceptability  at  each  node  of 
the  model  from  the  sub-dimensional  to  the  scenario  levels  were 
provided  to  the  SINCGARS  STF  for  each  iteration  of  the  model. 


Table  5-1 


A SUMMARY  OF  THE  MILITARY  VALUES  OF 
ALTERNATIVE  RADIO  SYSTEM  CONFIGURATIONS1 


Major  Dimensions  of  Value 

Alternative  Radio  System  Configurations 

Korean 

war 

Sys  tem 
(KWS) 

Current 

System 

(Alt.  1) 

Product- 

Improved 

(f?t8t!T 

Develop- 
mental 
System 
(Alt.  31 

Conceptual 

System 

(Alt.  41 

Operational  Acceptability 

. 32 

.69 

.73 

.74 

.80 

Technical  System  Utility 

.03 

. 34 

. 39 

.67 

.65 

Overall  Military  Value 

.17 

.61 

.56 

mui 

.72 

'Assuming  that  operational  acceptability  and  technical 
system  utility  are  weighted  equally  in  determining  overall 
military  value. 
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Table  5-2 

TECHNICAL  SYSTEM  UTILITY  AND  OPERATIONAL  ACCEPTABILITY 
BY  DIMENSIONS/SUB  DIMENSIONS  FOR  THE  KOREAN  WAR 
SYSTEM  (KWS)  AND  ALTERNATIVES  14 


Alternative 

KV;F 

Alt.  1 

Alt. 2 

Alt.  3 

Alt. a 

Technical  System  Utility 

.03 

. 34 

.39 

.67 

.65 

Dependability 

.08 

.49 

.53 

.79 

.78 

Technical  Performance 

.34 

.70 

.73 

. 84 

.83 

Receiver-Transmitter  Capability 

. 34 

. 80 

.79 

.88 

.86 

Retransmission  Capability 

.90 

.90 

.90 

.90 

.90 

Channel  Capabilities 

.23 

.76 

.83 

.92 

.97 

Interfact  Capabilities 

.19 

1.00 

1.00 

1.00 

1.00 

Technical  Flexibility 

.46 

. 37 

. 37 

.43 

.51 

Interchangeability 

.00 

. 00 

.00 

1.00 

.80 

Electromagnetic  Capability 

.67 

.70 

. 74 

. 52 

.51 

Operational  Acceptability 

.32 

. 69 

.73 

.74 

.80 

Operational  Flexibility 

.IS 

. 87 

.87 

.87 

.99 

Survivability 

.26 

.41 

.61 

.62 

.58 

Physical  Characteristics 

. 35 

.69 

.69 

.75 

.86 

Support  Requirements 

.77 

.87 

.37 

.88 

.89 

Transmission  Quality 

. 34 

.62 

.62 

.62 

.62 

Table  5-3 

TECHNICAL  SYSTEM  UTILITY  AND  OPERATIONAL  ACCEPTABILITY 
VALUES  FOR  ALTERNATIVES  14  AND  THE  KOREAN  WAR  SYSTEM 
(KWS)  AS  A FUNCTION  OF  PLATFORM 


ALTERNATIVE 

TECHNICAL  SYSTEM  UTILITY 

OPERATIONAL  ACCEPTABILITY 

Manpack 

Vehicular 

Airborne 

Manpack 

Vehicular 

Airborne 

KWS 

.05 

.03 

.00 

. 36 

36 

.29 

1 

.41 

.27 

. 34 

.68 

.64 

.77 

2 

.46 

.32 

. 37 

. 72 

.68 

CO 
1— • 

3 

.77 

.59 

.64 

.74 

.70 

.81 

.76 

. 56 

.66 

.85 

.73 

.82 

38 


-.hAUl  - L . p fc . 


On  the  basis  of  the  utility  values  displayed  in  Tables 
5-1,  5-2,  and  5-3,  it  is  clear  that  Alternatives  3 and  4 are 
far  superior  to  Alternatives  1 and  2.  Most  of  this  super- 
iority is  due  to  differences  in  technical  system  utility 
where  the  improvement  of  Alternative  3 over  the  current 
system,  Alternative  1,  is  greater  than  the  improvement  of 
the  current  system  over  the  Korean  War  System.  Alternative 
2 is  not  much  better  than  Alternative  1 with  respect  to 
either  technical  system  utility  or  operations]  acceptability. 

As  indicated  in  Table  5-2,  the  superiority  of  Alterna- 
tives 3 and  4 over  Alternatives  1 and  7 are  due  to  two  factors, 
namely : 

• The  greatly  improved  dependability  of  Alternatives 
3 and  4,  and  here  it  is  important  to  bear  in  mind 
that  technical  system  utility  is  a multiplicative 
combination  of  dependability  and  technical  perfor- 
mance. The  improvement  in  the  dependability  of 
Alternative  3 (and,  similarly,  Alternative  4)  over 
Alternative  1 is  about  50%  of  the  improvement  of  the 
current  system  (Alternative  1)  over  the  Korean  War 
System  (KWS) . As  prev  ously  indicated,  the  current 
system  reflects  the  technology  of  about  1960, 
whereas  the  KWS  reflects  the  technology  of  about  1950. 
The  improvement  of  Alternative  3 over  Alternative  1 

is  thus  quite  significant. 

• The  significant  improvement  in  the  technical  per- 
formance oJ  Alternatives  3 and  4.  The  improvement 
of  Alternai  ive  3 (and,  similarly.  Alternative  4) 
over  Alternative  1 with  respect  to  this  factor  is 
about  40  percent  of  the  improvement  of  the  current 
system  (Alternative  1)  over  the  KWS.  This  improve- 
ment, which  is  further  discussed  in  Section  5,1, 

is  due  to  improved  receive/transmit  capabilities, 
channel  capabilities,  and  interchangeability. 

The  differences  between  the  alternatives  is  not  as 
great  for  operational  acceptability.  With  respect  to  this 
dimension,  Alternative  4 is  better  than  Alternative  3,  the 
latter  being  essentially  equivalent  to  Alternative  2.  The 
superiority  of  Alternative  4 over  Alternative  1 is  due  to 
improved  technical  flexibility,  improved  survivability,  and 
superior  physical  characteristics.  The  improvement  of 
Alternative  4 over  Alternative  1 is  about  30  percent  of  the 
improvement  of  the  current  system  (Alternative  1)  over  the 
KWS  with  respect  to  operational  acceptability.  The  reasons 
for  these  differences  among  systems  with  respect  to  sub- 
dimensions of  technical  system  utility  and  operational 
acceptability  can  be  ascertained  by  examining  the  values  of 
the  systems  with  respect  to  the  technical  performance  charac- 
teristics that  describe  the  sub-dimensions.  This  was  done 
and  resulted  in  the  technical  conclusions  which  follow. 
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5.1  Technical  Conclusions 


• General  comparisons 

All  of  the  alternative  radio  systems  are 
essentially  equivalent  in  terms  of  retrans- 
mission capabilities. 

All  but  the  older  Korean  War  vintage  radio 
system  are  essentie lly  equivalent  in  terms  of 
interface  capabilities,  support  requirements, 
and  transmission  quality. 

The  alternative  systems  are  much  less  variable 
in  terms  of  operational  acceptability  than 
they  are  in  terms  of  technical  system  utility. 

There  is  little  change  in  the  relationships 
among  the  various  alternative  systems  from 
the  standpoint  of  scenarios  (FEBA  vs.  non- 
FEBA) . 

All  of  the  alternative  systems  are  poor  to 
unacceptable  with  respect  to  technical  flexi- 
bility. 

• Inter-system  comparisons.  Comparison  of  the 
alternative  radio  systems  in  pairs  from  the  oldest 
(Korean  War  vintage}  system  to  the  developmental 
an> l conceptual  designs  indicates  that: 

In  the  case  of  the  Korean  War  System  (KWS) 
vs.  System  1 (current  series  "of  radiosT: 

o System  1 is  obviously  far  superior  to 

the  Korean  War  System  in  terms  of  opera- 
tional acceptability  for  all  environments 
and  platforms.  More  specifically, 

System  1 is  superior  with  respect  to: 

Flexibility,  due  to  its  inter- 
operability and  COMSEC  capabilities. 
Survivability,  due  to  its  anti- 
jamming capabilities. 

Physical  characteristics,  due  to 
its  size,  weight,  form  factor,  and 
human  factors  engineering. 
Transmission  quality,  due  to  its 
secure  voice  transmission  capa- 
bilities . 

o System  1 is  superior  to  the  Korean  War 
System  in  terms  of  overall  technical 
system  utility.  More  specifically, 

System  1 is  superior  to  the  KWS  with 
respect  to: 
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Dependability,  which  is 
cative  dimension. 

Receive/transmit,  interface,  and 
channel  capabilities. 

o As  a result  of  the  foregoing  comparison, 
it  is  concluded  that  the  only  change  in 
importance  weights  that  could  make  the 
Korean  War  Bystem  superior  to  System  1 
would  be  to  put  all  of  the  weight  for 
technical  system  utility  on  technical 
flexibility.  The  I orean  War  System  is 
otherwise  a "dominc ted"  alternative  and 
therefore  will  not  be  compared  further 
with  the  other  alternative  systems. 

However,  it  does  serve  as  a good  bench- 
mark to  initiate  the  comparison  of  the 
performance  of  the  other  alternative 
systems . 

In  the  case  of  System  1 (current  series  of 

radios)  vs.  System  2 ( product- imprcved  ver'sion 

of  the  current  series  of  radios)  t 

o System  2 is  only  slightly  superior  to 
System  1 in  terms  of  both  operational 
acceptability  and  technical  system 
utility.  This  superiority  is  primarily 
due  its  dependability,  channel  capa- 
bilities, electromagnetic  compati- 
bility (EMC)  and  survivability. 

o As  a result  of  this  comparison,  it  is 

concluded  that  no  changes  in  the  impor- 
tance weights  or  utility  functions  could 
make  System  1 superior  to  System  2. 

Inasmuch  as  System  2 is  a product- 
improved  version  of  System  1,  it  is  only 
natural  that  System  1 is  a "dominated" 
alternative . 

In  the  case  of  System  2 (a  prod  ict- improved 

version  of  the  current  series  o'  radios)  vs. 

System  3 (a  developmental  radio  system)! 

o System  3 is  slightly  better  than  System 
2 in  terms  of  operational  acceptability, 
primarily  due  to  its  physical  charac- 
teristics of  size,  weight,  visual  de- 
tectability, and  human  factors  engineering. 

o System  3 is  far  superior  to  System  2 in 
terms  of  technical  system  utility, 
primarily  on  the  basis  of  the  depen- 
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dability  of  System  3 (a  dimension  which 
is  multiplied  with  technical  performance 
to  determine  overall  technical  system 
utility) . In  addition, 

The  overall  technical  performance 
of  System  3 is  superior  to  that  of 
System  2. 

The  receive/transmit  (R/T)  capa- 
bility of  System  3 is  superior  to 
'•  >f  System  2,  primarily  due  to 
r , communications  planning 

' • J ;•  iJt  ; ! • * r:  : . 

' . stem  j i:  t -<  • . :r-  rior  to  System 

2 with  respec'  ;:•?  1 capabilities 

for  an  airborne  (.-i  i . ,i'  and,  to  a 
lesser  degree,  with  respect,  to  the 
manpack  platform.  However,  this 
superiority  is  reversed  for  the 
vehicular  platform,  due  to  the 
number  of  preset  channels. 

System  3 is  more  technically  flexible 
and  is  superior  with  respect  to 
interchangeability. 

o System  2 is  superior  to  System  3 in 

terms  of  EMC,  primarily  du^  to  its  co- 
site and  co-channel  interference  capa- 
bilities . 


o In  order  to  make  System  2 superior  to 
System  3 with  respect  to  technics; 
system  utility,  an  extremely  heavy 
weight  would  have  to  be  assigned  to  r 
and  heavy  weights  would  also  have  to  be 
assigned  to  those  few  sub-dimensions 
wherein  System  2 is  occasionally  better 
than  System  3;  for  example,  as  previously 
cited,  channel  capabilities.  System  3 
completely  "dominates"  System  2 with 
respect  to  operational  acceptability. 

In  the  case  of  System  3 (a  developmental 
system)  vs . System  4 (a  conceptual  design) , 
tne  best  of  the  five  alternative  systems  are 
compared  and,  generally  speaking  (given  the 
current  structure/output  of  the  SINCGARS 
Evaluation  Model) , the  differences  between 
these  two  systems  appear  to  be  small.  More 
specifically,  however, 

w 

o System  4 is  slightly  superior  to  System 

3 with  respect  to  operational  acceptability, 
primarily  due  to  its  COMSEC  capabilities, 


its  manpack  and  vehicular  physical 
characteristics  (namely,  visual  detec- 
tability, form  factor,  and  human  factors 
engineering)  and  its  operational  flexi- 
bility. System  3,  on  the  other  hand,  is 
superior  to  System  4 with  respect  to  the 
survivability  (direction-finding  vulner- 
ability) of  its  airborne  radio. 

o System  3 appears  to  be  slightly  better 
than  System  4 with  respect  to  technical 
system  utility,  primarily  due  to  its 
manpack  CPR  and  interchangeability. 

System  4,  on  the  other  hand,  is  superior 
to  System  3 in  terms  of  the  R/T  and 
variable  power  output  capabilities  of 
its  airborne  radio  and  also  in  terms  of 
its  channel  capabilities  and  technical 
flexibility. 

System  3 could  be  made  greatly  superior  to 
System  4 in  terms  of  technical  system  utility 
by  heavily  weighting  the  R/T  capability.  On 
the  other  hand,  System  4 could  be  made  greatly 
superior  to  System  3 in  terms  of  technical 
system  utility  by  heavily  weighting  channel 
capabilities . 

With  respect  to  operational  acceptability, 

System  4 "dominates"  System  3 in  terms  of  all 
major  dimensions  except  survivability.  In 
order  to  reverse  this  superiority,  survivability 
would  have  to  receive  a very  heavy  weight  or 
the  weights  assigned  to  those  sub-dimensions 
wherein  System  3 is  occasionally  superior  to 
System  4 would  have  to  be  systematically 
changed  to  favor  System  3. 

Cost/Benefit  Comparison.  Combining  the  foregoing 
results  'of  the  evaluation  of  the  military  utility 
or  benefit  of  the  alternative  radio  systems  with 
the  results  of  the  evaluation  of  hardware  cost 
estimates  in  Section  4.0,  the  following  conclusions 
were  reached  with  regard  to  the  relative  cost/benefit 
merits  of  the  alternative  systems: 

Inasmuch  as  Alternative  2 was  far  more  expen- 
sive than  either  Alternative  3 ( r Alternative 
4,  the  latter  two  alternatives  i re  better  options 
from  a combined  cost/benefit  poi it  of  view. 

Because  Alternative  4 has  a lowe  hardware  cost 
that  Alternative  3,  and  the  military  values  of 
utilities  of  Alternative  3 and  4 are  approxi- 
mately equal.  Alternative  4 is  considered  to  be 
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the  most  cost/beneficial  option.  However,  this 
conclusion  must  be  qualif ied  somewhat  by  the 
fact  that  the  employment  of  two  receiver/trans- 
mitters with  Alternative  3,  rather  than  an 
auxiliary  receiver,  increases  its  average  unit 
cost  ($AUC)  by  about  $500.00.  If  this  situation 
were  rectified,  Alternative  3 would  compare 
quite  favorably  with  Alternative  4 from  a cost/ 
benefit  standpoint. 

5 . 2 Methodological  Conclusions . 

5.2.1  General.  The  SINCGARS  evaluation  model  las 
served  to  successfully  differentiate  among  the  four  given 
alternatives  and  has  shown  that  two  are  clearly  superior. 
Furthermore,  since  the  model  is  public  with  a definite 
structure,  the  reasons  for  the  superiority  of  the  systems 
are  made  clear  by  the  model.  In  addition,  "what  if'  analyses 
can  be  conducted  to  ascertain  how  changes  in  performance 
characteristics  will  lead  to  changes  in  utility.  This 
allows  not  only  the  evaluation  of  other  feasible  SINCGARS 
alternatives  not  currently  under  consideration , but  also  the 
manipulation  of  various  aspects  of  the  conceptual  system. 

To  that  end,  the  SINCGARS  staff  has  been  provided  with  a 
computer  terminal  which  allows  then  to  conduct  additional 
"what  if"  analyses  when  they  so  choose. 

Thus  the  SINCGARS  Evaluation  Model  shows  where 
changes  in  performance  lead  to  the  greatest  changes  in 
utility.  Knowing  the  costs  associated  with  such  changes, 
the  user  can  address  not  only  the  evaluation  problem,  but 
also  the  design  problem.  Evaluation  models  such  as  that 
used  for  this  SINCGARS  evaluation  allow  evaluation  and 
design  to  proceed  simultaneously.  This  is  achieved  by 
answering  the  general  evaluation  question,  "What  should  a 
good  radio  system  do?"  as  opposed  to  the  more  specific 
question,  "What  will  these  specific  alternatives  do?"  Yet, 
at  the  same  time,  the  output  of  an  evaluation  mo  lei  allows 
specific  recommendations  with  respect  to  the  chc  Ice  of 
alternatives  to  be  made — as  was  the  case  with  the  SINCGARS 
Evaluation  Model. 

There  are  methodological  implications  of  using 
an  approach  such  as  that  which  was  used  in  the  SINCGARS 
evaluation.  One  implication  is  that  the  structure  of  the 
model  will  differ  somewhat  from  one  which  evaluates  only 
predefined  specific  alternatives.  The  latter  will  not 
contain  factors  for  which  all  systems  are  known  to  be  the 
same.  This  is  because  the  utilities  for  those  factors  will 
be  equal  across  alternatives  and  will  essentially  be  additive 
constants  in  the  final  overall  utilities.  It  is  argued  ;hat 
such  constants  obscure  the  "important"  differences.  However, 
when  considering  design  problems  where  the  alternatives  are 
not  completely  certain,  such  factors  must  be  included.  The 
reason  is  that  even  on  those  dimensions  where  all  systems 
are  initially  equivalent,  the  alternrtives  can  change.  For 
example,  the  conceptual  system  could  be  improved  relative  to 
the  others,  or  a^new  system  could  be  introduced. 
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5.2.2  Specific  implications  with  respect  to  ranges  of 
technical  performance  characteristics.  The  inclusion  of  the 
foregoing  factors  has  implications  for  the  ranges  of  techni- 
cal performance  characteristics  over  which  utility  functions 
will  be  assessed.  When  the  alternatives  are  completely 
specified  and  will  not  change,  the  range  of  a technical 
performance  characteristic  is  from  the  worst  value  with 
respect  to  that  characteristic  among  the  alternatives  under 
consideration  to  the  best  value  among  those  alternatives. 

When  design  considerations  enter  the  picture  and  when  "what 
if"  analyses  are  to  be  conducted  on  an  interactive  basis, 
the  range  must  be  from  the  mir  .mum  feasible  to  maximum 
feasible  values  of  that  characteristic. 

Close  attention  must  thus  be  paid  to  the  outputs 
at  all  levels  of  evaluation  models.  The  inclusion  of  all 
factors  relevant  to  utility  implies  that  weights,  utility 
functions,  and  even  the  structure  are  independent  of  the 
options  under  consideration.  Utilities  are  relative  only  to 
some  standard,  as  was  true  in  the  SINCGARS  evaluation  where 
that  standard  was  the  Korean  War  radio.  Sine*  the  systems 
can  be  equivalent  with  respect  to  many  factors  of  the  model, 
the  utility  values  for  the  alternatives  are  likely  to  be 
closer  than  they  would  be  if  only  the  specific  systems  were 
being  evaluated. 

5.2.3  Specific  implications  with  respect  to  scenario 
usage . A second  methodological  implication  involves  the  use 
of  scenarios  for  system  evaluation.  Experts,  when  defining 
scenarios  as  conditioning  variables  that  can  cause  variation 
in  the  performance  of  systems,  usually  choose  variables 
which  lead  to  variations  in  system  performance  but  do  not 
discriminate  between  systems  because  all  systems  vary  in 
relatively  the  same  manner  with  respect  to  these  variables. 
Close  attention  must  thus  be  given  to  the  scenario  definition 
process.  Scenarios  which  simultaneously  accomplish  the 
following  two  goals  are  desirable: 

• They  must  allow  the  evaluation  of  the  expected 
utilities  of  the  alternative  systems. 

• They  must  also  discriminate  among  the  various 
systems  and  thus  identify  special  design  issues. 

These  goals  are  rarely  compatible  and  each  has  implications 
for  scenario  design.  When  these  goals  are  incompatible,  the 
user  must  decide  which  route  to  take  with  respect  to  scenarios. 
If  the  main  goal  is  evaluation,  broad  representative  scenarios, 
-uch  as  those  used  in  this  SINCGARS  evaluation,  allow  calcu- 
lation of  overall  expected  util  ty.  When  design  is  the 
major  consideration,  as  was  not  the  case  in  this  SINCGARS 
evaluation,  hig  ily  specific  scenarios  must  be  used  to  dis- 
criminate betwe jn  alternative  designs  and  to  locate  optimum 
configurations . 

As  a result  of  the  evaluative  questions  generated 
during  this  SINCGARS  evaluation,  as  well  as  during  other 
recent  projects,  close  attention  should  be  given  to  the 
scenario  usage  problem. 
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This  report  also  reflects  the  development  and  utilization  of  an 
interactive  Monte  Carlo  program  for  the  evaluation  of  alternative  hardware 
cost  estimates. 

Background  and  Approach.  The  objectives  of  the  research  described 
in  this  report  were  to: 

• Develop  and  test  a multi-attribute  utility  model  which  was 
used  to  assist  the  SINCGARS  STF  in  evaluating  the  military 
utility  of  alternative  Combat  Net  Radio  configurations.  Of 
particular  importance  during  the  development  of  the  model  was 
the  need  to  provide  an  analytic  tool  which  would  accept  "last 
minute"  changes  in  data  and  would  also  enable  a user  to  conduct 
"what  if"  analyses  on  an  interactive  basis.  The  multi-attribute 
utility  model  which  was  developed  utilizes  the  technical 
performance  characteristics  of  the  radio  systems  as  inputs. 

• Develop  and  apply  an  interactive  Monte  Carlo  program  for 
evaluating  the  hardware  cost  estimates  of  the  alternative 
systems. 

A hierarchical  structure  for  the  multi-attribu  :e  utility  model 
consisting  of  several  levels  was  developed,  startin’,  with  military 
utility  and  partitioning  this  into  major  dimensions  of  utility,  e.g., 
technical  system  utility  and  operational  acceptability.  These  dimen- 
sions were  further  sub-divided  into  sub-dimensions  tfhich,  in  turn,  were 
further  partitioned — each  partition  becoming  more  specific  until  a level 
was  reached  at  which  one  or  more  technical  performance  characteristics 
served  to  describe  each  of  the  sub-dimensions.  The  military  utility  for 
different  levels  of  each  of  the  performance  characteristics  was  estab- 
lished by  assessing  a utility  function  over  the  relevant  range  of  that 
characteristic.  The  relative  importances  of  the  different  performance 
characteristics  were  assessed  by  assigning  relative  importance  weights 
to  the  components  of  each  level  of  the  hierarchy.  The  procedures  for 
assignment  depended  upon  how  the  components  combine  to  determine  system 
utility  at  the  level  in  question.  Combination  rules  (additive  or  multi- 
plicative) were  then  applied  to  aggregate  the  utilities  for  the  components 
both  within  and  across  the  various  levels  of  the  hierarchy. 

The  resultant  structure  thus  systematically  combined  expert  judgment 
and  technical  performance  characteristics  to  provide  a model  which 
accurately  aggregated  the  actual  measures  of  the  performance  characteristics 
of  a particular  system  so  as  to  yield  a measure  of  the  military  utility 
of  the  system. 

The  Monte  Carlo  program  was  used  to  develop  a probability  distribution 
for  the  $Al'C  of  each  of  the  radios  and  its  ancillary  components.  These 
were  then  combined  to  yield  probability  distributions  for  the  overall 
$AUCs  by  platform  for  each  alternative  system. 

Findings  and  Implications.  Having  structured  and  tested  the  SINCGARS 
Evaluation  Model  as  previously  described,  the  model  was  used  to  evaluate 
four  alternative  radio  systems,  as  well  as  a Korean  War  System  (KWS) 
which  served  as  a benchmark  for  comparison  of  utilities.  As  a result  of 
this  evaluation,  two  alternative  systems  were  identified  as  Deing  superior 
to  the  others  from  the  standpoint  of  military  utility.  Similarly, 
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utilizing  the  Monte  Carlo  program,  one  of  the  alternative  systems  was 
identified  as  being  the  least  expensive  in  terms  of  hardware  costs. 
Combining  the  foregoing  results  of  the  evaluation  of  the  military  utility 
or  benefit  of  the  alternative  radio  systems  with  the  results  of  the 
evaluation  of  hardware  cost  estimates,  one  of  the  alternative  radio 
systems  was  identified  as  the  best  cost/beneflclal  option. 

The  two  most  important  methodological  implications  for  the  application 
of  decision  analysis  which  were  derived  from  the  foregoing  evaluation 
involve  appropriate  definition  of  the: 

e Range  of  technical  performance  characteristics  to  be  used  In 
the  analysis. 

e Scenarios  for  design  and  evaluation  decisions. 
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